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Abstract 

A linear group G < GL{n) acts on d-tuples of n x n matrices by simultaneous conjugation. 
In [Adv. Math. 19 (1976), 306-381] Procesi established generators and relations between them 
for G-invariants, where G is GL{n), 0{n), and Sp{n) and the characteristic of base field is 
zero. We continue generalization of the mentioned results to the case of positive characteristic 
originated by Donkin in [Invent. Math. 110 (1992), 389-401]. We investigate relations between 
generators for 0(n)-invariants. 

2000 Mathematics Subject Classification: 16R30; 13A50. 

Key words: invariant theory, classical linear groups, polynomial identities. 

1 Introduction 

1.1 Relations 

We work over an infinite field F of arbitrary characteristic charF. All vector spaces, algebras, and 
modules are over F unless otherwise stated. By algebra we always mean an associative algebra. 
We consider the ring of polynomials 

R = Rn= F[a;.y (fc) 1 1 < i, j < n, 1 < fc < d], 

in n^d variables. It is convenient to organize these variables in so-called generic matrices 

(Xii{k) ••• Xln{k) \ 
■ 
(fc) ... (fc) / 

Denote coefficients in the characteristic polynomial of an n x n matrix X by at{X)^ i.e., 

n 

det{X + XE)^J2 A""*crt(X). (1) 

So, (70 (X) = 1, ai(X) = tr{X) and cr„(X) = det{X). 

Let G be a group from the list GL{n), 0{n), Sp{n), where we assume that charF 7^ 2 in case 
G — 0{n). The algebra of matrix G-invariants is the subalgebra of R generated by at (A), 
where 1 < t < n and A ranges over all monomials in 
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. Xi,...,Xd, if G = GL(n); 



. Xi,...,Xd, Xl,...,Xj, if G = 0(n); 

• Xi, . . . , Xd, XI, . . . , X^, if G = Sp{n). (Here X* stands for the symplectic transpose of X). 

Moreover, we can assume that A is primitive, i.e., is not equal to the power of a shorter monomial. 
If the characteristic of F is zero, then it is enough to take traces instead of Cf , 1 < t < n, in order 
to obtain i?*^. 

For a field of characteristic zero generators for matrix invariants of G G {GL{n), 0{n), Sp{n)} 
were described by Sibirskii in [TO] and Procesi in [T7]. Generators for G — SO{n) were calculated 
by Aslaksen et al. in [5]. Over a field of arbitrary characteristic generators for matrix GL{n)- 
invariants were found by Donkin in [5]. In [21] Zubkov described generators for matrix 0{n)- and 
i^(n)-invariants, where in the orthogonal case he assumed that charF ^ 2. Generators for matrix 
<S'0(n)-invariants were calculated by Lopatin in [T5] . 

In case of characteristic zero relations between generators for matrix GL{n)-, 0{n)- and Sp{n)- 
invariants were established by Procesi in [T7]. Independently, relations for _R'^^(") were found by 
Razmyslov in |18] . Over a field of arbitrary characteristic relations for matrix GL(n)-invariants 
were established by Zubkov in [20] (see also Theorem II. 5p . 

In this paper we describe relations for matrix 0(n)-invariants modulo free relations in case 
charF ^ 2 (see Theorem II. ip . As a corollary, we explicitly establish relations between indecom- 
posable invariants (see Corollary 16. 6p , where an element is called decomposable if it belongs to 
F-span of products of homogeneous invariants of positive degree. 

The above mentioned systems of generators are infinite, but it can be showed that they contain 
finite systems of generators. Moreover, the goal of the constructive theory of invariants is to find 
out a minimal (by inclusion) homogeneous system of generators of i4r[y]*^ explicitly. It is an 
important problem, which arose as early as the theory of invariants itself. To find out a minimal 
system of generators we need a description of relation only between indecomposable invariants. And 
Corollary 16.61 gives us such description for i?*^*^"). As an application, in the upcoming paper |16j 
some estimations are given on the highest degree of elements of a minimal system of generators 
for (ggg Theorem O below). 

For more information on finite generating systems for matrix invariants see overviews [5] and [S] 
by Formanek. For recent developments in characteristic zero case see [7] and in positive charac- 
teristic case see [3]. For results concerning generators and relations between them for mixed 
representations of quivers see survey |14| . 

Note that it is possible to define 0{n) and SO{n) in characteristic two case. But in this case 
even generators for invariants of several vectors are not known (for the latest developments see [4]). 

We introduce the following notions. 

• Let M be the monoid (without unity) freely generated by letters 1, . . . ,d,l^ , . . . ,(f" . 

• Let TV C .M be the subset of primitive elements, where the notion of a primitive element is 
defined as above. 

• Let A4f be the vector space with the basis A4. 

Assume that a = ai ■ ■ ■ ap and /3 are elements of Ai, where ai, . . . , are letters. 
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• Introduce an involution on Ai as follows. Define /3^'^ = /3 for a letter /3 and — • • • of € 
M.. We extend the introduced map to an involution : M.^ — >■ M.¥ by linearity. 

• We say that a and /3 are equivalent and write a ~ /3 if there exists a cyclic permutation 
TV G Sp such that a7r(i) ■ • ■ a7r(p) = /? or a7r(i) ■ " ■ Q^7r(p) = 

• Let Ada- be a ring with unity of (commutative) polynomials over F freely generated by 
"symbolic" elements o't(a), where t > and a S A^f- 

• Let Afa be a ring with unity of (commutative) polynomials over F freely generated by "sym- 
bolic" elements at{a), where t > and a G A/" ranges over ^-equivalence classes. We can also 
define Afa as a factor of Aia by some ideal (see Lemma [3. ip . In particular, we can consider 
at{a) with a € AAw as an element oiMa- 

We will use the following convention: 

tr(Q;) = (yi{a) 
for any a G M.¥- For a letter fi G M. define 

Xf, if/3 = *^ ■ 

Given a = ai • • ■ ofp G where Ui is a letter, 
Consider a surjective homomorphism 

defined by at{a) — > at{Xa), if i < n, and (Tt{a) otherwise. Its kernel if„ is the ideal of 
relations for i?'^^"). Elements of nj>o called free relations. For a,/?, 7 G Aiw and t,r G N, 

an element at^rict, PtI) S A/'o- is defined in Section [3] (see Definition 13. 2p . where N stands for the 
set of non-negative integers. 

Theorem 1.1. //charF ^ 2, then the ideal of relations Kn for i?*^*^") ~ Ma/ Kn is generated by 

(a) at^rict, /3, 7), where t + 2r > n and a, /?, 7 G Air; 

(b) free relations. 

Since F is infinite, in the formulation of Theorem ll.il we can take the following elements 

(a') at.r.,s{.xi, . . . ,Xu,yi, ■ ■ ■ ,yv,zi, . . . , Zyf) for t + 2r > n, where t = {ti, . .. ,tu) G N", r = 
(ri,...,r„) GN", s= (si,...,s^) G N"", t <i + ■ ■ • + t„, r = n + ■ • • + r„ = si + • • • + .s^, 
and xi, . . . , z„ G 

instead of (a), where at.r.s is given by Definition 14.11 

In Lemma 16.51 we show that the ideal of free relations lays in the ideal of Afa generated by 
at{ay^^'-'^ , where a ranges over J\f and t > 0. The following conjecture is valid in characteristic 
zero case. 

Conjecture 1.2. In case G — 0{n) there are no non-zero free relations. 
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For f € R denote by mdeg/ its multidegree, i.e., mdeg/ — {ti,. . . ,td) £ N'^, where tk is the 
total degree of the polynomial / in Xij{k), 1 < i, j < n. The algebra R^'-"^ is homogeneous with 
respect to N'^-grading. Denote the degree of a G 7W by deg a, the degree of a in a letter P by 
deg^ a, and the multidegree of a by 

mdeg(Q:) = (degi a + deg^r a,..., deg^ a + deg^T a). 

For t > we assume that degCTt(a) — tdega and mdeg at (a) — t mdeg a. Therefore TV, M, 
and 7V(j have N-gradings by degrees and N''-gradings by multidegrees. Note that the given above 
generating system for i?*^*^"^ as well as relations (a') are N'^-homogeneous. 

1.2 The known results on relations 

In this section we explicitly formulate the results on relations that have been mentioned in Section[T] 
Let us remark that modulo free relations. Theorem 11.11 generalizes Theorem ll.3l in the same manner 
as Theorem 11.51 generalizes Theorem 11.41 

Theorem 1.3. (Procesi flT^ ) //charF = 0, then the ideal of relations Kn for R^'-"^ ~ J\fa/Kn is 
generated by <Jt,r{a., /?, 7), where t + 2r = n + \ and a, /?, 7 £ M.w- 

We denote by M.' the monoid (without unity) freely generated by letters 1, . . . ,ci and denote 
by A/"' C M' the subset of primitive elements. We define M'^ {M'^, respectively) similarly to 
Mw {Ma, respectively). Let Af^ be a ring of (commutative) polynomials over F freely generated 
by "symbolic" elements at (a), where t > and a e A/"' ranges over all primitive cycles (i.e., 
equivalence classes with respect to cyclic permutations) . An analogue of Lemma 13.11 is also valid 
for Afa- Hence we can consider at{a) with a G M'^ as an element of JV^. 

Theorem 1.4. (Razmyslov \18^ . Procesi \17l ) //charF = 0, then the ideal of relations if'^ for 
j^GL(n) r^j^i^jx'^ generated by CT„+i(a), where a G M^. 

Theorem 1.5. (Zubkov, 1201) The ideal of relations K'^ for i?<3-^(") ~ M'^jK'^ is generated by 
Otipi), where t > n and a G M'^- 

Remark 1.6. In case G = GL{n) there are no non-zero free relations (see [6]). 



1.3 Historical remarks on at^r 

The complete linearization of (Jt.r was introduced by Procesi (see [l7j. Section 8 of Part I), where 
it was denoted by Fk,n+i- It is not difficult to see that 

CrJ'"(a;i, . . . ,Xt,?/i, . . . . ■.,Zr)= Fr.t+2r{yi, ■ ■ ■ , Vr, Xl , ■ ■ ■ , Xt, Zi, . . . , z^) in JVa, (2) 

where crj'" stands for the complete linearization of at,r (see Definition 14. ip and xi, . . . ,Zr G A^f. 
Then at^r was introduced by Zubkov in [21]. Note that the definition from [2J is different from 
our definition and their equivalence is established in Lemma 17.141 
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Another way to define at^r is via the determinant-pfafiian iyPr,r{X, Y, Z) that was introduced 
in [13] as a "mixture" of the determinant of X and pfaffians of Y and Z (see Example 17.51) . 
Lemma 14.91 and Example 17.51 imply that DP^.r relates to at^r in the same way as the determinant 
relates to it, i.e., for n x n matrices X, Y, Z we have 

to 

BPrAX + XE, Y,Z)^Y1 A*''"Vt,,(X, r, Z), (3) 

where n =^ to + 2r and to > 0. Since <Ttfi{X,Y, Z) = (Jt{X) (see Remark 13. 4p . Formula ([3]) turns 
into ([1]) for r = 0. Note that this approach gives us (7t^r{X,Y, Z) as a polynomial in entries of 
matrices X, Y, Z. But for our purposes we have to present at^r{X, Y, Z) in a different way, namely, 
as a polynomial in at{a), where t ranges over positive integers and a ranges over monomials in 

x,y,z,xt,yt,zt. 



1.4 The structure of the paper 

The paper is organized as follows. In Section [2] notations are given that are used throughout the 
paper. Key notion at^r is introduced in Section[3l In Section|4]we obtain some results on at^r- In 
Lemma l4. 21 a formula for partial linearization of crt,r is presented, which is similar to Formula ()10|) 
from the definition of at,r- In characteristic zero case Lemma 14.41 allows us to work with complete 
linearization of at,r instead of at,r itself. 

In Section [5] Theorem 11.11 is proven. The proof is based on results from paper [53] by Zubkov, 
where an approach for computation of relations for i?*-^^") was given (see Sections 3 and 4 of [24]). To 
complete Zubkov's approach we used the decomposition formula from [12] , which can be considered 
as a generalization of Amitsur's formula for at^r- 

In Section |6| we establish some restrictions on free relations (see Lemma 16. 5|) . In particular, 
we show that there is no non-zero linear free relations between indecomposable invariants (see 
Corollarv l6.6p . 

In Section [7| the decomposition formula is formulated (see ((2T|) ). Then it is shown that our 
definition of at,r coincides with the original definition from [24] (see Lemma r7.14|) . Since Section [7| 
contains numerous notions that have limited usage in the paper, we put it at the end of the paper. 



2 Notations 

In what follows, Q stands for the quotient field of the ring of integers Z. For a vector t = 
{ti, . . . ,tu) G N" we write t\ = ii! • • • The length of p-tuple c = (ci, . . . , Cp) we denote by 
#c = p. For short, we write 1^ for (1, . . . , 1) e W. We assume that stands for the identity 
m X m matrix. 

Consider n,k > 0. If k < n, then we write ^^"^ binomial coefficient; otherwise, we 

define We also define ^ ^ =0. Let us remark that =1. 

By substitution 1 ai, . . . ,d ad in a E M, where ai, . . . , G A4, we mean the substi- 
tution 1 ^ ai,...,d — ?> ttd, 1^ — aj,...,(f" — >■ and denote it by a\i-^ai,....d^aa- Similar 
convention we also use for substitutions of elements of Af^ and so on. 
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Denote by ;S = ^[xi, . . . , Xm] the polynomial ring in rci, . . . , Xm over F, i.e., S is a commutative 
F-algebra with unity generated by algebraically independent elements xi,. . . ,Xm- Given an N- 
graded algebra A = J2i>o ^^'^ ^ ~ Sj>o '^i ^ '^here almost all ai € Ai are zero, we write 
hc*(^) for J2o<i<s ^^'-^ hcs(a) = ai for the homogeneous component of degree s. 

A quiver Q= {Qo,Qi) is a finite oriented graph, where Qo is the set of vertices and Qi is the 
set of arrows. Multiple arrows and loops in Q are allowed. For an arrow a, denote by a' its head 
and by a" its tail, i.e., 

a 

-@ ■ 

We say that a = ai ■ ■ ■ as is a path in Q (where ai, . . . , as G Qi), if a" = aj, . . . , a's-i — Q^s, i-e., 




The head of the path a is a' = a[ and the tail is a" — a". A path a is called closed if a' = a" . A 
closed path a is called incident to a vertex v € Qq ii a' = v. Similarly, closed paths . . . , /J^ in 
Q are called incident to v ii 13[ = ~ (3'^ — v. 

Definition 2.1 {of a mixed quiver). A quiver Q is called mixed if there are two maps ■ Qa ^ Qo 
and ^ : Qi — > Qi such that 

, = V, (3^^ = (3- 

for all u e Qo and /3 G Qi. 



Assume that Q is a mixed quiver. Denote by A^(Q) the set of all closed paths in Q and denote 
by Af{Q) C A4{Q) the subset of primitive paths. Given a path a in Q, we define the path a'^ and 
introduce ^-equivalence on M{Q) in the same way as in Section [1] Moreover, we define Mr{Q), 
MaiQ), and N(t{Q) in the same way as TWf, Ma, and Na have been defined in Section [T] The 
notions of degree and multidegree for elements of 7W(Q) and Nct{Q) are introduced as above. 



Example 2.2. Consider the quiver Q: 




21,^1 ,-22,^2 



where there are four arrows from vertex 1 to vertex 2, there are four arrows in the opposite 
direction, and each of its vertices has two loops. Here letters Xi,xJ, yi,yf, Zi,zf {i — 1,2) stand 
for arrows of Q. We define 1-^ = 2 for vertex 1, so Q is a mixed quiver. Then 

t T \T T T T T T T T T 

• (xiyi ziY = z{ yix{ , yizi ^ y{ zi , xiyixf zi ^ xiy{ xi z{ ; 

• deg(?/i2:f) = 2, Aegy^{yizi) = 1, deg^^r (yizi) = 0, and Ta.Aeg{xiXiX2yixJ x'^ z'^) = 
(3,2, 1,0, 0,1). 
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3 The definition of at^r 

In this section we assume that ^ is a commutative unitary algebra over the field F and all matrices 
are considered over A. 

Let us recall some formulas. In what follows A, Ai, . . . , Ap stand for n x n matrices and 
1 < t < n. Amitsur's formula states [T]: 

aMi + • • ■ + Ap) = ■ ■ ■ (4) 

where the sum ranges over all pairwise different primitive cycles 71, . . . , 7^ in letters Ai, . . . , Ap 
and positive integers ji,---,jq with J2i^ijidegji = t. Denote the right hand side of ^ by 
Ft,p{Ai, . . . , Ap). As an example, 

(72(^1 + A2) = a2iAi) + (73(^2) + tr(Ai) tr(A2) - tr(v4iA2). (5) 

Note that for a G ^ we have 

at{aA) = aVt(^). (6) 
For I > 2 we have the following well-known formula: 

ii,...,iti>0 

where we assume that ai{A) = for i > n. Denote the right hand side of ([7|) by Pt^i{A). In ^ co- 
efficients 6-*''^ e Z do not depend on A and n. If we take a diagonal matrix A = diag(ai, . . . , a„), 
then (Tt(^') is a symmetric polynomial in ai, . . . ,a„ and ai{A) is the z*^ elementary symmetric 
polynomial in oi, . . . , a„, where 1 < i < n. Thus the coefficients j with tl < n can easily be 
found. As an example, 

tTiA^)=triA)^ -2a2{A). (8) 



Lemma 3.1. We have Afa — Aia/ L for the ideal L generated by 

(a) at{al^ V Up) ~ Ft^p{ai, . . . ,ap), 

(h) at{aa) - a^-at^a), 

(c) at{a')-Puia), 

(d) atia/S) - (Jtifia), 

(e) at{a) - (7t[a^), 

where p > 1, a, ai, . . . , ttp G Aiw, a €E F, t > 0, and I > 1. 

Proof. Consider a homomorphism p : M„ ^ J\fa- defined by fit (a) /, where we apply equalities 
(a), (b), and (c) one after another to at{a), a £ A^f, to obtain /. Obviously, p is well defined. 
Since p is surjective, to complete the proof it is enough to show that p{f ) = 0, where f E L ranges 
over elements (a)-(e). 
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Assume that / is not element (e) and p(/) ^ 0. We define M' ^ M'„^ and A/"^ in the same way 
as in Section [TT2] with the only difference that now we start with the monoid M' freely generated 
by letters 1, . . . , Id. Note that M.' ~ A^, where the isomorphism is given by i — i and i + d ^ 
for all 1 < i < d. Hence ~ M.^ and / can be considered as an element of J^'„. Define 
p' : M'„ — 7> TV^ in the same way as p. Since p{f) ^ 0, we have p'{f) ^ 0. But p'{f) is a free 
relation for i?*^^*^"); a contradiction to Remark 11.61 

We assume that / is element (e), i.e., 

s s 
1=1 i=l 

where G F and ai & M. Denote the result of application of (a) and (b) to / by /i G Ma- Let 

h^Yl {a,, (^W) . . . a,, (7(^)) - a,, (7P') ■ • ■ a,, (7^ ')), 

(9) 

where the sum ranges over ji,...,jq and cycles 71,..., 7^ in letters Xi,...,Xs as in (jj]), 7;^^'' 
is the result of substitution xi ai,...,Xs cx.s in 7;, and 7; is the result of substitution 
a;i ^ , . . . , Xs in 7; [l <l <q)- Note that for any pair (j, 7) from ^ there exists a pair 

(j,^) from jSl) such that 7,^^^ - for all 1 < ^ < g. Applying (c) to (Jnilf^) ■ ■ •crj,(79^^) and 

("2) ('2) 

CTji ) ■ ■ • cTj^j we obtain equal elements in Af^- Thus, = 0. □ 

Let r e N. In order to define Ot^r, we consider the quiver Q 




where there are two arrows from vertex 1 to vertex 2 and there are two arrows in the opposite 
direction. We define — 2 for vertex 1 to turn Q into a mixed quiver. 

Definition 3.2 {of (Tt.r{x,y, z j). Denote hy I — It,r the set of pairs {j,a) such that 

• =f/=j = =ffa = p for some p; 

• ai, . . . , ttp e J^{Q) belong to pairwise different ^-equivalence classes and ji, . . . ,jp > 1; 

• ji mdeg(Q;i) H h jp mdeg(Q;p) = {t, r, r). 

Then we define at^rix, y, z) G Na{Q) by 

at^r{x,y,z)= ^ (-!)« crj-,(ai)---crjjap), (10) 

(i,i3L)6l 

where p = #j = H=a and — S,j,g_~ t + Y^^i=i Ji(degy a; + deg^ ai + 1). For t = r = we define 
(70,0(2;, y, 2) = 1. Moreover, 
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• if a, /3, 7 G A^F, then we define (Jt^r(pi, 7) G ■N'a as the result of substitution x ^ a, y — ?> /?, 
z 7 in ([T0| : 

• if X. F, Z are n x n matrices, then we define at^r{X, Y, Z) ^ A as the result of substitution 
X ^ X , y ^ Y, z ^ Z in pHt . where we assume that <Jj{A) — foi t > n and any n x n 
matrix A. 

Example 3.3. 1. If t = and r = 1, then ^-equivalence classes on J\f{Q) are yz, yz"^ , . . . Hence, 
ao^i{x,y,z) = -tT{yz) +tr(yz^). 

2. If f = r = 1, then ^^-equivalence classes on Af{Q) are 

X, yz, yz'^, xyz, xyz'^, xy'^z, xy'^z'^, . . . 
and we can see that ai^i{x, y, z) — 

— tr(x) tr(j;z) + tr(a;) tv{yz'^) A- tv(xyz) — tvixyz'^) — tr(xy'^z) -f tr(xy'^z'^). 

Remark 3.4. atfl{x,y,z) = <Jt{x). 



4 Properties of 

In this section all matrices have entries in a commutative unitary algebra A over the field F. 

Definition 4.1 (0/ a partial linearization of at^r)- We assume that t = G N", r = 

(ri, . . . ,r,) G N", s = (si, . . . , s„) G N"" satisfy 

si H h Si, = ri -I h r.u,, 

and xi, . . . , x„, j/i, . . . , j/t,, zi, . . . , Zi(, belong to Mr. Consider at^r{aixi + • • ■ + a^Xu, ^ij/i + • • ■ + 
) G A/'o- as a polynomial in cii , . . . , G , 61 , . . . , 6^ , ci , . . . , , where fxi , . . . , c-^; G 
F. We denote the coefficient of ■ ■ ■ a*^bl^ ■ ■ - bl^cl^ ■ ■ ■ c^" in this polynomial by 

o't,r,s(a;i, . . . ,Xu,yi, ■ ■ ■ ,yv,zi, . . . , z^) g Afa- (11) 

In other words, this coefficient is equal to the homogeneous component of at.r{xi + ■ • • + a^u, yi + 
■ ■ • + yy, Zi + • ■ • + Zyj) of multidegree {t,r,s). In multilinear case we have v = w and write 

^u.v {■^i J • ■ • I Xu tUI, ■ ■ ■ tUv, Zi, . . . , Zyj) 

for ai^^i^^i^ (xi, . . . , a;„, j/i, . . . , ?/^, Zi, . . . , Zi(,), where the definition of 1" was given in Section (3] 

Given for n x n matrices Xi, . . . , Z^, we define at^r,s{Xi, . . . , X„, Yi, . . . ,Yy, Zi, . . . , Z^,) G »4 
as the result of substitution x\ — > X\ , . . . , z^j — >■ Z^, in element (|lip of TVo- and use the assumption 
that (Tj(j4) = for j > n and any nx n matrix A. 
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In this section we will show that an analogue of Formula pUj) from the definition of at,r is valid 
for (Jt^r^s (see Lemma [4.21 below). Generalize the construction from Section [3] as follows. Using 
notation from Definition 14. 1[ we define the mixed quiver Qt,r,s- 




where there are 2w arrows from vertex 1 to vertex 2, there are 2v arrows in the opposite direction, 
and 1^ = 2. Here by abuse of notation xi,. . . ,Ztn stand for the arrows of Qt,r,s as well as for 
elements of A4r from the definition of at^r- Hence we can assume that Ncr{Qt.r,s) C Ncr- 

Given a path a of Qt,r,s, denote the total degree of a in xi, . . . , Xu by deg^ a, and the total 
degree of a in x\ ^ . . . , a;^ by deg^-T a. Similar notation we also use for y and z. As an example, 
Aeg^T{xiyixJx2z'^) = 2. 

Assume that X — Tt,r,s is the set of pairs {j,a) such that 

• = =ffa = p for some p; 

• ai,...,ap € N'{Qt,r,s) are representatives of pairwise different '^-equivalence classes and 
ji,---,jp > 1; 

• jimdeg(ai) H hipmdeg(ap) = {t,r,s). 

Lemma 4.2. We have 

'^t,r,s{xi, ...,Xu,yi,-. ■,yv,Zi, ...,Znj)= ^ (-1)'' crii(ai) • • -CTj-plap), 

where p ^ 4j^j_ = and rj = T]j_,a = ii H h tu + Y7i=i J»(degy + deg^ + 1). 

Proof. We set t — ii + ■ ■ • + t„ and r = ri + ■ ■ ■ + r^ = si + • • • + s^. By definition, 
(^t,r, s{xi-, ■■■ ,Xu,yiT ■■■ ^yv, zi, Zw) is equal to the homogeneous component of multidegree 
{t,L,S.) of 

/ = <yt,r{xi H ^ Xu,yi-\ h J/u, Zl H ^ Zyj) 

where ^, ai, . . . ,ap, and ji, . . . ,jp are the same as in the definition of at^s- Let Q be the quiver 
from Section |31 For a closed path a in Q denote the result of substitution a; — > ^ , y -> ^ j/j , 
z -I- X] -Zfc in a by 

where is a closed path in Qt.r,s for all I. As an example, the result of substitution x ^ xi+ X2, 
y yi+ y2, and z ^ Zi + Z2 in yz^ is equal to yizf + yiz^ + 2/2^1" + 2/2^2"- 
Applying Amitsur's formula (j4]) to /, we obtain 

/ = E(-l)'' ■ (-1)^' ^.11 (711) • --^n,, (7i,J * • • • * {-l)^^^jAlpi) ■ ■ •'^.P,,(7p,J, (12) 
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where = ji — {jn + • ■ • + iigj- Here for any 1 < i < p the sum ranges over pairwise differ- 
ent primitive cycles jn,. . . ,jiq- in letters af^ (where I € L{ai)) and jn, . . . ,jiq^ > 1 such that 

jii deg(7ii) H \-jiq deg(7ig, ) = ji deg(ai). 

We claim that taking the homogeneous component of / of multidegree {t, r, s) we obtain the 
required formula for crt,r,8- We split the proof into several statements. 

1. We have (-l)«+«i+-+«p = {-If, where v = t + ^Li Efe=i jik{degy{jik) + deg,{jik) + 1). 
Proof. For 1 < i < p we have deg^^ = deg^^ af^ for all I. Then 

, / X , deg7ife 
deg,(7..) = deg,a,.^^ 

for all k. Therefore 

ji degy ai = ^^^{ja deg(7ii) + • ■ • + iig deg(7igj) = 

= jn degj^(7a) H \- jig degyij^g^). 

The same formula is also valid for z. Now it is easy to see that the required formula is valid. 

Given a closed path 7 in Q/_,r,s, denote the result of substitution xi,. . .,Xu x, yi,. ■ .,yv — >■ 2/, 
and 2i, . . . , — > z in 7 by ^(7)- We can consider tpi-y) as a closed path in Q. Let us remark that 
if = for a, ^ G M{Q), p,q > 0, then there exists a 7 e M{Q) such that a = 7' and P = 7-' 
for some in particular, if (3 is primitive, then a = /3* for some i. 

2. Let a be a primitive closed path in Q and 7 6e a primitive word in letters a^'^, where I € 
L{a). Then j is a primitive closed path in Qt,r,s- In particular, 711, . . . , 71^^ , . . . , jpi, . . . , 'jpg^ are 
primitive closed paths. 

Proof. Assume that there exist fc > 1 and a primitive closed path /? in Qt,r,s such that j = (3'^. 
We have 7 = a^'^^^ ■ ■ ■ a''™) for some h,. . . ,lm & L{a). Thus a™ = ^{P)'^ and applying the above 
mentioned remark we obtain that there exists a j > such that ip{l3) = . Since dega^'^ = dega 
for all I, we have /3 = a^'^' ■ • • a^'^\ Hence, 

7= (a('^)...a('^))*= = a('^) 

Closed paths a^'^ (where I G L{a)) are pairwise different. So the last formula implies that k = U+j 
for any i < m — j. In other words, 7 is not a primitive word in letters a^'\ where I G L{a); a 
contradiction. 

3. Closed paths 711 , ... , 7igj^ , • • • , 7pi , • ■ • . Jpgp are pairwise different. 

Proof. Assume that 74^ = 7jm- Equalities ip{lik) = cij • • • aj and '(piljm) = CKj " • ' j together 
with V(7ifc) = i^iljm) imply that i = j (see the above mentioned remark). Since 7^1, ... , jig. are 
pairwise different, we have k = m. 

To complete the proof of the lemma we apply substitution ip in the same way as in the proof 
of Statement 3. □ 
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Denote by T]l"^ the set of a such that {l^,a_) G Xik .^j ii. , where p = ^a. The following corollary 
is a trivial consequence of Lemma 14.21 



Corollary 4.3. We have 

<jt'^y{xi,...,Xu,yi,-.-,yv,zi,...,Zy) = ^ (-l)'-tr(a), 

where 77^ — w + X^f^iC^egj, + deg^ ai + 1), p = #a, and tr(a) stands for tr(ai) • • • tr(ap). 
Lemma 4.4. Assume that F = Q. Then for x,y,z ^ A^p we have 

t r r 

Proof. In this proof we use notions formulated in Section [7] (see below). Consider the tableau 
with substitution (T, {X, Y, Z)) of dimension {t + 2r,t + 2r) from Example 17.21 By Lemma [TH we 
have 

crt,r{x,y,z) = Sgn(Cj,c) crji(ci) ■ • •CTjp(Cp)|i^:r, 2^j,, 3^z, 

(2,c)eiT 

where p = #j — #c. 

Using Lemma 4 from [12], we can reformulate Theorem 3 from [12] for (7~, {X, Y, Z)) as follows: 

if n = t + 2r, then 

^[7^ E ^g^(^) n t^(^^w) = E sgn(0,e)a,,(XeJ---a,jX,J, (13) 

where p = = fj^c. Consider the result of formal substitution X ^ 1, F — 2, and Z — > 3 
in (ini), where 1, 2, 3 £ are letters: 

fH)2 E ^g^(^) n ^^('^(q)) = E sgn(fj,c) crji(ci)---crjjcp). (14) 

We claim that (fT4|) is valid equality of two elements from Af^ . Theorem 3 was proven in Sections 5, 
6, and 7 of [12]. We repeat this proof without using Section 5, i.e., we do not apply Lemma 4 from 
Section 5 in the reasoning several lines before Formula (14) from [l^ . Since 7" has two columns, 
we obtain the claim. Lemma 17.121 concludes the proof. □ 

Lemma 4.5. Assume that F = Q. Then at,r,s{xi, ■ ■ ■ , Xu, yi, . . .yv, zi, . . . , Zw) = 
1 



(J^ ^(Xi , . . . , , . . . , Xu '^ui Ul-i • • ■ 1 yi-i • • • Uv -i • • • -i Vv t • • • • ■ • 1 1 ■ • • 1 ^w)-) 



t}. r\ s\ 

ti t„ j-i 

where t = ti + ■ ■ ■ + and r = ri + • • ■ + r„ = si + 
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Proof. The left hand side of the required formula is equal to the homogeneous component of 

o-t,r{xi H \-Xu,yi-{ hy ) of multidegree (i,r,s). Applying Lemma l44l to at,r 

and using linearity of ctJ'", we complete the proof. □ 

Lemma 4.6. Assume that Xi, . . . ,Xt, Yi, . . . , Yr, Zi, . . . , are n x n matrices and there exists 
a j such that Yj or Zj is a symmetric matrix. Then 

'''t'r(^lj ■ • • 1 ^t,Yi, . . . , Yr, Zi, . . . , Zr) = 0. 

Proof. Assume that Yj is a symmetric matrix. Consider a primitive closed path a — (3yj in 
Si',!'-,!"- such that deg^ a + deg^x a < 1 for any arrow 7 from Qim^,i>-. Then ayj is also a 
primitive closed path and ayJ ^ ayj . Lemma 14.21 concludes the proof. □ 

Let us recall that i?„ stands for the identity n x n matrix. 

Lemma 4.7. Let Xi, . . . , Xt^i, Yi, . . . , Yr, Z\, . . . ,Zr be n x n matrices. Then 

'''t'r(^li • • • I ^t~l, En,Yi, . . . ,Yr, Zi, . . . , Zr) = O CrJ'"]^ (Xi , . . . , Xf_i, Yi, . . . ,Yr , Zi, . . . , Zr) , 

where a = n ^ (t + 2r) + 1. 

Proof. Denote Xt = En and /i ~ ctJ'"(Xi, . . . , Xt, Yi, . . . , Yr, Z\, . . . , Zr). Let x\, . . . ,Zr be arrows 
of Qi*,!'-,!'-. Denote by the substitution Xi Xi, yj — > Yj, and Zj — > Zj for all 1 < i < t and 
1 < j ^ ^- By Corollary 14.31 we have 

Ai= ^ (-l)''^V'(tr(a)) =Mi+M2, 

where /zi is the sum of all ( — l)''^tr(a) such that aj ~ Xt for some j. Equalities 

Mi=tr(S„) (-l)"^'/'(tr(^)) and 

M2 = -(i + 2r-l) (-l)''^^(tr(^)) 

conclude the proof. □ 

The following remark is trivial. 

Remark 4.8. Let p > be the characteristic of F and F[a;i, . . . , Xm] be a polynomial ring. Define 
the ring homomorphism tt : Z — >■ F by 7r(l) = If, where If stands for the unity of F. Consider 
f — "^2 '^ifi £ . . . , Xm], where Oi G 7r(Z), and fi is a monomial in xi, . . . , Xm for all i. Take 



13 



bi G {0, 1, ... ,p — 1} C Z such that 7r(6i) = and set h = J^i ^ifi G Q[''Ci, • • • , a^m]- Then h = Q 
implies / = 0. 



Lemma 4.9. Let X,Y, Z be n x n matrices and n > t + 2r. Then 

at AX + XE,,,Y, = ^ [ " " + ^'^^ + * ] y <Jt-,AX, Y, Z) 

for any X G A. 

Proof. By Remark 14.81 without loss of generality we can assume F = Q. Then Lemma [4.51 and 
linearity of crj™ imply 

at AX + A£;„, Y, Z) ^ E ( ! ) ^' al';(X,..^.,X, K,...,£;„ , Y, . . . ,Y, Z, . . . , Z). 

t—i i 

Lemma [4.71 concludes the proof. □ 



5 Proof of Theorem 11.11 

Without loss of generality we can assume that F is an algebraically closed field (cf. Remark 2.1 
from [22] and Remark 3.2 from 21 ). 

Assume that Xi = X„^i, . . . , Xd = Xn^d, Y ^ Yn, Z = Zn are nx n generic matrices and entries 
of these matrices are Xij{l), . . . ,Xij{d), yij, Zij (1 < i,j < n), respectively. Consider the mixed 
quiver Q 




where there are d loops in each of its vertices and A = 2. 

Given a G A4{Q) we define Xn^a in the same manner as in Section (T] using convention that 

{Xna, a a = Xi 
Yn, a a^y . 
Zn, ifa = z 

Denote by C„ — ¥[xij{k), yij, 1 1 < «, j < n, 1 < fc < d] the coordinate ring of the space of 
mixed representations of Q of dimension vector {n,n). Then its algebra of invariants Jn C C„ is 
generated by at{Xn.a), where 1 < t < n and a is a primitive closed path in Q (see [23]). We define 
the inclusion _R„ C C'„ in the natural way. Consider a surjective homomorphism 

defined by crf(a) — > at{Xn.a), if t < n, and at{a) — ?> otherwise. Its kernel T„ is the ideal of 
relations for J„. Elements of CI^^q Ti are called free relations for J„. 
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Let (a, 7) be a triple of paths in Q. Then it is cahed good if a' — a" = 1, /3 is a path from 2 
to 1, and 7 is a path from 1 to 2. Since Lemma |3. II can be reformulated for J\fa{Q)^ we have 

i^t,r(^ auaii, ^ a2ja2j, ^ asfeasfc) e Ma{G), (15) 

i j k 

where aii,a2j, a^k G IF and [an, a2j, asfe) is a good triple for all i,j, k. 

Theorem 5.1. (Zubkov, 124^ ) The ideal of relations T„ for J„ :^ J\fa-{G) /T„ is generated by 

• elements il5\) for t + 2r > n; 

• free relations for Jn ■ 

Given N > n, we define two homomorphisms PN,mPN,n '■ C'at ~^ C'n as follows: 

PN,n ■ ^N,k -t n h^JV-^ n n '^^V^ 

o n / ' "ii^ 



where \ < k < d and Ejsi-n stands for the identity matrix. In other words, we assume that 



PN,n{Xij{k)) = 



Xij{k), if 1 < i, J < ?^ 
0, otherwise 



and so on. For short, we use the following notations: p — pN,n and p — pN,n- 
Consider a mapping : Jn Jn defined by: 

for all 1 < k < d. The following diagram is commutative: 

MAG) Jn t^^RO(N)^_Eii j^^ 



p 



p=p 




The next lemma is a reformulation of Lemmas 3.3 and 3.4 from 



Lemma 5.2. If N > s > 0, then 

the kernel of ^ : hc''(7V(j) — > hc'*(i?'^'^^)) is generated by free relations for i?*^*^"^ of degree 



s; 



the kernel of Tat : he" {Ma{Q)) hc'*( Jat) is generated by free relations for J„ of degree s. 

15 



Lemma [5.21 implies that we can identify hc''(7Vcr) with hc''(i?'^(^)) as weh as hc*(A/'o-(t7)) with 
hc''(JAr) modulo free relations. Let us formulate Proposition 3.2 from j24] . 

Lemma 5.3. The ideal Kn is generated by elements ^j^{hf), where 

• f ranges over elements 115\} with t + 2r > n; 

• N > deg/ = s and N ^ n is big enough, so we can assume that f G J^; 

• h — hf(z Jn such that pih) — 0, deg /i = s, and hCs(/i) = hCs(/). 
Here ^N^hf) is considered as an element of Ma- 



Note that in Lemma 15.31 f can be non- homogeneous. 

Given t, r > satisfying t + 2r > n, we consider an element p5|) 

/ = <^t,r{fl,f2, fs), 

where /i — ^jai^aii, /2 — J2j ^'^j'^'^j^ ^^'^ = ^k'^^kct^k- There exists an iV ^ n such that 
N > t + 2r, N > t + n, and we can assume that / G J^. In what follows we will write /i instead 
of aiiXN^a-^. and similarly for /2 and /a. 
By Theorem 15. li we have 

p(/) = 0. (16) 

But we can not claim that p{f) — 0. We will construct h — hf G Cat that is "close" to / and 
p{h) — (see Lemma [53] below) . 

We can rewrite /i as /i — fu + /12, where /n is a sum of all auXN^an such that an contains 
an arrow Xk or xj^ for some k. Similarly, we can rewrite /2, /s as /2 — /21 + /22 and /a = /31 + /32, 

T 

summands of fi^2 for I — 1,2,3. Note that 



where /21 and /31 contain all summands with Xk or xT. Let A; G F be the sum of coefRcients of 



p{fl)=pUl) + XlEN,n (17) 

for / = 1,2,3. Here EN,n stands for the following N x N matrix 



-EjV-ri 



Lemma 5.4. 



(/)=E(^Z ")p(^*-.,.(/i,/2,/3)) Al. 



p 

i=0 

Proof. By Remark |4.8[ without loss of generality we can assume F = Q. By (IT7|) . we have 

P{f) = crt,r(i)(/l) + AiE'AT^n, p(/2) + \2EN,n, Pifs) + Aai^AT,™ )■ 

Lemmas 14.41 14.61 together with linearity of crj™ imply p{f) = 

E ^«:"( P(/1),---,P(/1); i^iV,n,. . .,ii;Ar.n , p(/2), . . . ,p(/2), Pifs), • . • ,p(/3)) Al . 
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Since p{fi)EN^n = for all I, applying Lemma 14.21 we obtain 
1 * 1 

P(^) = W ^ ,,(^_,), • • ■ ,P(/i),P(/2), . . . ,p(/2),p(/3), . • . ,p{f3)) Ala, 

where a = a]'g(i?jv,ra, ■ • ■ , -E'w.n)- By Lemma and Remark we have 

a = i!cri, o(-E^Ar,n, 0,0) = i!cr,;(i;Ar,„) = i! ^ ^ j 

Lemma [4.41 concludes the proof. □ 

Lemma 5.5. There exist 69, • ■ • , £ ^ stic/i t/iai 

t 

^ ^ ^/ X! biX{at^i^r{fi, f2, /a) e -^AT 

i=0 

satisfies the following conditions: 

a) p{h) = 0; 

b) degh — s and hcs{h) — hcs{f), where s — degf. 
Proof. We set bo — 1. By Lemma [5.41 we have 

t t — i / \ 

p{h) = E ^1 E 7 ) p{<^t-^-,AflJ2J3)) Ai. 

n o — n V / 



We substitute k ior t — i — j and obtain 

t t-k 



fe=0 i=0 

The following system of linear equations 

t-k 

t — i — k 



N -n 
t — i — k 



E ^« ( • \ ) = 0, where < fc < i (18) 



i=0 



with respect to &i, . . . , 6f is triangular and has 1 on the main diagonal. Hence this system has a 
solution &i, . . . , fet G Z. Using Formula we can see that the equality = holds. Obviously, 
deg/i = s and hcs{h) = hcs{b()at^r{fij2, fa)) = hcs(/). □ 

Lemma 5.6. Assume that bo = 1 and 61, . . . , 64 satisfy system of linear equations U8\) . Then 



t-k 



4=0 
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for0<k<l<n-2r. 

Proof. Denote the left hand side of the required formula by bk,i- 

The proof is by decreasing induction on I. li I = n — 2r, then b^^i = by p^ . 
Let I < n — 2r. Then formula 



where < p < q, implies 




We substitute j for i — 1 and obtain bk,i — bk,i+i + bk+i,i+i- Induction hypothesis concludes the 
proof. □ 

Denote by In the ideal of R'^'-^^ generated by crp.,q{Y,^ cuX{-fu), Y,j C2jX{^2j), Y.k CikX{^zk)), 
where ^(7) stands for X^^-y, p + 2q > n, ju, 72^,73*; are monomials in xi, . . . , a;^, x^, . . . , xj, and 

Cu,C2j,C3k & F. 

Lemma 5.7. Assume that h G Jn is the element from Lemma \5.5\ Then $jv(/i) belongs to In. 

Proof. For short, denote $Ar(//i) = gi for 1 — 1,2, 3. We have $Ar(/;) — gi + XiEn for all I. 

To begin with, we assume that F = Q. Using Lemmas 14.41 and 14.61 together with linearity of 
cr}'" in the same manner as in the proof of Lemma 15. 4[ we obtain 

i 

^N{h) = '^biX\at-i^r{gi + AiE'at, 32,53)- 

i=0 

Lemma 14.91 implies 

<^N{h) = ^ ak,r (51 ,92.93) \{-' E f^-^'l + ^'A. (19) 

k=0 i=0 ^ ^ 

In the general case Remark 14.81 shows that Formula (IT5|) is also valid. 
U k + 2r < n, then 

i:t.-,-.('"-<*'+^^')=o (20) 

by Lemma \5M Thus <&Ar(ft-) belongs to In. D 

Lemma 5.8. Any element g = at,riJ2i'^iilii, J2j'^'2jj2j, J2k'^3kj3k) ofAf^, where t + 2r > n and 
cu,C2j,C3k e 7ij,72j,73fe e M, belongs to K^. 

Proof. There exists a g' G Ma{g) such that ^n{^n{9')) = *Ar(5)- Since T„(g') = by 
Theorem 15. 11 we obtain 

^n{9) = V ° *w(.9) = P o o Tn{9') = o T„(5') = 0. 

□ 

Lemmas 15.31 15.71 and 15.81 conclude the proof of Theorem 11.11 
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6 Free relations 

In this section it is convenient to write xi, . . . , Xd, xj , . . . , for letters 1, . . . ,d, l"'" , . . . ,(f" that 
are free generators o{ ^A. Similarly to Section[l] we denote by A^°° the monoid freely generated by 
letters xi,X2, ■ ■ ■, xf ,X2 , ■ ■ ■ and define JV^ {-^^ , respectively) similarly to JV (A/'a, respectively). 

Definition 6.1 (o/Lin(/)). Given N'^-homogeneous / S Afa with mdeg/ = t= {ti, . . . ,td), we 
define its complete linearization Lin(/) G as follows. Let h G Af^ be the result of substitution 
Xi — > X]*Lo '^ij^i+jd (1 < i < d) in /, where a^- G F, and we consider /i as a polynomial in Oy . 
We denote the coefficient of an • • • ■ • • aai ■ ■ ■ adta ^ by Lin(/). 

Definition 6.2 (o/ Lin^^(a)). Given a G A4°°, we define Lin^^(Q;) G as the result of substi- 
tution Xij^jd Xi {1 < i < d and j > 0) in a. 

Example 6.3. If / = (T2{xi), then (O implies that Lin(/) = — tr(.i:ia;d+i) + tr(a;i) tr(xc;+i). 
If / = tr(a;i)^, then Lin(/) = 6 tr(xi) tr(.T<j+i) tr(a;2d+i)- 

Given / = aj-^ (ai) ■ ■ ■ aj^iap) G J\fa-, where ai, ... ,ap G A/", we define c/ and e/ as follows. We 
consider a subset S of the set of pairs {ji,ai), I < i < p, such that elements of 5' are pairwise 
different with respect to ~, where (ji,ai) ~ ijk,ctk) if and only if ji — jk and at ^ a^. Given 
{j,a) G S, we denote C(j^q) = #{(ji,aj) | Ji = j,a, a,l < i < p}. We set c/ = I\(j,a)es 
and Cf = p. 

The proof of the following lemma is similar to the proof of Lemma 14.21 



Lemma 6.4. Let f = Uj-^ (ai) • • • <Tj^ {oip) G Afa, where ai, . . . , Op G A/", and t — mdeg /. Then 
Lin(/) = {-iy^+-+^''-^ Cf 5]tr(7i)...tr(7p)+c//i, 

where 

• 7i, . . . ,7p G M°° are representatives of pairwise different '^-equivalence classes; 

, , / \ 1 / N f ^1 if I = i + jd for 1 < i < d and < j < ti 

. we have deg,, (71 • • • 7p)+deg,. (71 • • • 7p) = | q, otherw^sc 

for all I > 0; 

• Lin^"'^(7i) — oP^ for all I < i < p. 

• h — ±/ifc, where is a product of p + I or more traces. 



Lemma 6.5. //charF > 0, then the ideal of free relations lays in the ideal of Ma generated by 
at{aY^^^''^ , where a ranges over Af and t > 0. //charF = 0, then the only free relation is trivial. 

Proof. Consider an N'^-homogeneous free relation / G Afa- 

1. Let mdeg f — t satisfy ti < 1 for all 1 < i < d. We assume that / 7^ 0. Then / = J^k ^kfk, 
where ak G F, Ok ^ 0, and fk are pairwise different products of traces. Denote n — deg / and denote 
by eij the nxn matrix whose («, j)**^ entry is 1 and any other entry is 0. Let /i = tr(Q!i) • • • tr(ap) 
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for some ai, . . . , ap € TV. Given ai — yi • ■ • Uq, where yi, . . . ,yq are letters, we set Yi — e^.i+i for 
1 < i < q and Yq — Cq^i. Considering a2, . . . , ap, we define li for q < i < deg/i. Since / is a 
free relation, the substitution yi —> Yi [1 < i < deg/i) implies that ai = 0; a contradiction. Thus, 
/ = 0. 

2. For some q we have / = a^/i + where Oj, hj G F, a^, 6j ^ 0, /i, /ij are pairwise 

different products of at {t > 0) such that ef- — q and eu^ > q for all Since Lin(/) is also a free 
relation, linearity of Lin, Lemma 16.41 and part 1 of the proof imply that c/^ =0. Hence fi lays in 
the required ideal of Afa in case charF > and / = in Afa in case charF = 0. Since Lin(/i) = 0, 
we can repeat the same reasoning for h — bjhj and so on. □ 

Given an N-graded algebra A, denote by A'^ the subalgebra generated by elements of A of 
positive degree. It is easy to see that a set {ai} C ^ is a minimal (by inclusion) homogeneous 
system of generators (m.h.s.g.) for A if and only if {oi} is a basis for A = A/{A'^)'^ and {ai} 
are homogeneous. Let us recall that an element a € ^ is called decomposable if it belongs to the 
ideal {A'^)'^. Therefore the least upper bound for the degrees of elements of a m.h.s.g. for i?*^*^"' is 
equal to the maximal degree of indecomposable invariants and we denote it by Umax. Theorem ll.il 
together with Lemma 16.51 imply the following corollary. 



Corollary 6.6. //charF 2, then the ideal of relations Kn of R^^"-^ ^JVa/Kn is generated by 
at,r{ot, /3, 7), where t + 2r > n and a, /3, 7 range over A4w- 

As an application of Corollarv l6.6l we obtained the following result in 1161. 

Theorem 6.7. Let n — 3 and d > 1. Then 

• If charF = 3, then 2d + 4 < Anax <2d+7. 

• // charF 7^ 2, 3, then D^i^^ — 6. 

As about matrix GL(n)-invariants in case n — 3, its minimal system of generators was explicitly 
calculated in [TUI and [llj . 

7 Appendix: other definitions of cr^ ^ 

In this section we assume that ^ is a commutative unitary algebra over the field F and all matrices 
are considered over A. In what follows we recall some definitions from [T^]. Note that in this 
section we consider only rectangular tableaux with two columns whereas in 12 tableaux with 
arbitrary number of columns of any length were defined. 

Definition 7.1 {of a tableau with substitution). A pair (7", (A'l, . . . ,Xd)) is called a tableau with 
substitution of dimension {n,n), if 

• T is a rectangular tableau with two columns and n rows. The tableau T is filled with arrows 
in such a way that an arrow goes from one cell of the tableau into another one, and each cell 
of the tableau is either the head or the tail of one and only one arrow. We write a G T for 
an arrow a from 7". Given an arrow a gT, denote by a' and a" the columns containing the 
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head and the tail of a, respectively. Similarly, denote by 'a the row containing the head of 
a, and denote by "a the row containing the tail of a. Schematically this is depicted as 

a" a' 







7 

a 





• ip is a fixed mapping from the set of arrows of T onto {1, . . . , d} that satisfies the following 
property: 

if a, 6 e T and ip{a) — (p{b), then a' — b' , a" = 6"; 

• Xi , . . . , Xd are n x n matrices. 

Example 7.2. Let X, Y, Z he {t + 2r) x (t + 2r) matrices and let T = Tt,r be the tableau: 



xi- 








xt- 




yi 


Zl 


-t 


\ 








yr 


Z 


r 




1 


h 



We define f as follows: 'p{xi) — 1, ^p{yj) — 2, ip{zj) = 3 for 1 < i < i and 1 < j < r. Then 
(7~, 1^, Z)) is a tableau with substitution of dimension [t + 2r, t + 2r). 

Example 7.3. Let Xi, . . . , Xt, Yi,...,Yr, Zi . . . , Zr he {t + 2r) x (t + 2r) matrices and let T be 
the tableau from Example 17.21 We define ip as follows: ip{xi) — i, (p(jjj) — t + j, ip{zj) ^ t + r + j 
for 1 < i < t and 1 < j < r. Then (T, {Xi, . . . , X*, Yi, . . . , i;, Zi . . . , Z,.)) is a tableau with 
substitution of dimension (< + 2r,t + 2r) . 

Definition 7.4 (o/bpf7-(Xi, . . . , X^)). Let Xi, . . . , be n x rt matrices and let (T, (^i, . . . , Xd)) 
he a tableau with substitution of dimension (n, n). Then we define 

h^i-T-{Xi,...,Xd) = ^sgn(7ri)sgn(7r2) W_{X^(a))^^n("a),7T^,('a), 
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where {Xip(^a))ij stands for (i, j)*^ entry of the matrix X^^^^a) a-nd the sum ranges over permutations 
7ri,7r2 S Sn such that for any a,b ^ T the conditions ip{a) = ip{b) and "a < "b imply that 
TTjC'a) < TTiC'b) for i — a" ~ b" . For F = Q there exists a more convenient formula 

bpf^(Xi, . . . = — ^ Sgn(7ri)sgn(7r2) J]^ (X^(Q))^^„("a),7r„,('a)- 



Example 7.5. Assume that (T, Z)) is the tableau with substitution from Example 17.21 

Then 

DP,,,(X,r,Z) =bpf^(X,y,Z), 
where the determinant-pfaffian DPj.,r(^, Y, Z) was introduced in [13j . 

The decomposition formula was formulated in Theorem 3, [T^ (see (pij) below). Assume that 
(T, {X, Y, Z)) is the tableau with substitution from Example l7.2l Denote by dec7-(X, Y, Z) the right 
hand side of the decomposition formula, applied to the tableau with substitution (T, {X,Y, Z)). 
Given x, y, z G A^f, we assume that dec7-(a;, y, z) e J\f^ stands for the result of formal substitution 
X ^ X, Y ^ y, and Z ^ z in decr{X, Y, Z). 

Lemma 7.6. We have ut.rix, y, z) — dec7-(x, y, z) for x,y, z G A4w, where T — Tt.r is defined in 
Example\T^ 

In particular, at,r{X, Y, Z) ^ DPr^r(X, Y, Z) for {t + 2r) x (t + 2r) matrices X, F, Z. 
To prove this lemma we need we need additional definitions from [12|. 

Let (T, {Xi, . . . ,Xd)) be a tableau with substitution of dimension {n,n). Consider Ai°° and 
from Section[6l where letters are 1,2,.. ., 1-^, 2^, . . . Given a e T, we consider (p{a) e {1, . . . ,d} 
as an element of A^°°. For m G define the matrix Xu in the same way as in Section [TJ Here 
we assume that Xi = X^r = for i > d. 

For an arrow a ^ T denote by a-^ the transpose arrow, i.e., by definition {a^)" = a' , {a^)' — a" , 

T 

"{cL^) = 'a, '{a^) = "a, and ^p{a^) = <p(a)"^ G A4°°. We write a G T if a is an arrow or a transpose 
arrow of T. 

T 

Definition 7.7 {of paths). We say that oi, 02 G 7" are successive in 7", if a'^ ^ and ' a\ — "02. 

A word a = ai • • • fls, where ai, . . . , G T , is called a path in 7", if a^, a,;+i are successive for 
any 1 < i < s — 1. In this case by definition ^{a) = <^(ai) • • • (/?(as) G M°° and = • • ■ is a 
path in T; we denote a!^,' as,a!{," a\, respectively, by a' ,'a,a" ,"a, respectively. 

A path ai • • • fls is closed if a^, ai are successive. 

T 

Consider words a — ai • • • a^, 6 = b\- ■ - bq, where oi • • • a^, bi ■ ■ - bg G T (in particular, both 
a, & might be paths in T). We write a & if there is a cyclic permutation tt £ Sp such that 
a^(i) • • •a^(p) is equal to b or 6"^. We will use similar notations for elements of M°°, for sets of 
paths, and for subsets of A^°°. 
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Denote by Td a set of representatives of closed paths in T with respect to the ~-equivalence. 
Example 7.8. Let 



a - 






( 


b 


c 



be a fragment of tableau. Then a and h(F are closed paths. 

Definition 7.9 [ofT'^). Given r S S'„, we permute the cells of the 2"*^ column of T by r and 
denote the resulting tableau by . The arrows of T"^ are {a^ \ a € T}, where ip{a^) = ^(a), 
(a^)" = a", (a^)' = a', and 



'{an 



"a, if a" = 1 
T("a), if a" = 2 



'{an 



'a, if a' = 1 
T('a), if a' = 2 " 



Obviously, (T"^, {Xi, . . . , Xa)) is a tableau with substitution. For short we will write instead 

of (r-)ci. 

We use notation {. . .}m for multisets, i.e., given an equivalence = on a set S and 
ai, . . . ,ap,bi, . . . ,bq £ S, we write {ai, . . . , ap}m = {bi, ■ ■ ■ , bq}m if and only if p = q and 



#{1 <j<p\aj= tti} = #{1 <j<p\bj= Ui} 



for any 1 < i < p. 



Definition 7.10 {of If). Let j E W and c = (ci, . . . , Cp), where ci, . . . , Cp £ arc primitive 

and pairwise different with respect to ~ . Then (j,c) is called a T-pair. 

A T-pair (j,c) is called T -admissible if for some ^ = ^j,c € •S'n the following equivalence of 
multisets holds: 

VC^l) ^ {^1' • • • , Cl, . . . , Cp, . . . , Cp}m- 
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3p 



We write (j ,c^) (i,c) and say that these pairs are equivalent if and only if 

{Ci, . . . , C]^ , . . . , Cp, . . . , C„}m ~ {ci , . . . , Cl , . . . , Cp, . . . , Cp^m- 

V ' V ' ^ „ ' 
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3p 



If {j^,cP_} ^ (j, c) and (j,c) is T-admissible, then the pair {j^,^ also has the same property, since 
we can take ^^0 ^0 = ^j^c- Denote by I7- a set of representatives of T-admissible pairs with respect 
to the ~-equivalence. 



The decomposition formula states 



bpf 7-(Xi ,...,Xd)= ^SHk'^) '^31 (-^ci ) • • • (^u (^cp ) , 

(j,c)eir 



(21) 
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where p = = =ffc. Note that ^j^c is not unique for a representative of the ^-equivalence class 
of a T- admissible pair (j, c), but sgn(^j_c) is unique (see Lemma 17.111 below) . Hence 

decr{x,y,z)^ ^ fign{^j^^c) crjii^^i) ■ ■ ■ <^3p(cp)\i^x, 2^y, 3^z (22) 
(i,c)Glr 

for x,y, z € A4w- 

Proof of Lemma \7. 6] Given an arrow or transpose arrow a of T, we can consider (p{a) € A4°° as 
an arrow of the quiver Q from Section [3] as follows: 1^5(2;^), <-p{xJ) are loops in vertices 1 and 2 of 
Q, respectively, f{yj) goes from vertex 2 to vertex 1 and so on. Similarly, for any word a in arrows 
and transpose arrows of T we can consider ip{a) as a word in arrows of Q. Note that 

• if a is a path in 7", then 1^9(0) is a path in Q; 

• if a £ Tci, then ip{a) is a closed path in Q. 

Hence for any T-admissible pair (j, c) we can consider ci, . . . ,Cp as closed paths in Q. Therefore it 
is not difficult to see that Ij- ~ It.r- Formula together with Lemma r7.11l (see below) concludes 
the proof. □ 

Lemma 7.11. We use notations from Lemma \ 1. 6\ and its proof. For any (j,c) G I7- we have 

sgn(0,c) = (-1)«, 

where S, = t + X^itfi Ji(degy Ci + deg^ Ci + 1). Ci is considered as a path in Q; therefore degy ci 

and deg^ a are well defined. 

Proof. We start the proof with the definition. For a permutation tt G St+2r and a = ai ■ ■ ■ Og G T^Yi 
where oi, . . . , Os are arrows or transpose arrows of 7", there is a permutation t G St+2r such that 

• T(i) = i for any i G {1, . . . , t + 2r}\{'ai, . . . , 'cs}; 

• for any 1 < i < s there exists a 6 G T such that a\ and 6 coincides, i.e., (a[)" = 6", (a[)' = 6', 
"< = "6, '< = '6. 

Denote sgn(a) — sgn(r). In spite of non- uniqueness of r, sgn(a) is well defined, it does not depend 

on TT, and sgn(a) = sgn(a-^). For any 1 < i < #j we consider ai G such that (piai) = Ci. 

Then we have 

*i 

sgn(0,c) = n«gnK)''- (23) 

i=l 

Arrows of the tableau T are Xi, yj, Zj, where 1 < i < t and 1 < j < r. Let tt G St+2r, o G 7^^, 
and let h, c be some paths in . 

1. If a Xi, then sgn(a) = 1. 

2. If a = bxiC, then sgn(a) — — sgn(xi) sgn(&c) = — sgn(6c). 
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3. If a = UjZk, then sgn(a) = — 1. 

4. If a = Ujz'^ , then sgn(a) = 1. 

5. If a = byjZkC, then sgn(a) = — sgn{yjZk) sgn(6c) — sgn(6c). 

6. If a = byjz'^c, then sgn(a) — — sgn(yjZ^) sgn(6c) — — sgn(&c). 

7. If a = hzkVjC, then sgn(a) = - sgn(t/jZfe) sgn(6c) = sgn(6c). 

8. If a = bz'^yjc, then sgn(a) — — sgn(yjZ^) sgn(6c) — — sgn(6c). 
For a € we have two possibihties: 

• If degy a + deg^ a = 0, then sgn(a) = (-l)d°gx a+dcg^T a+i ^g^g ^^^^^ ^ 2). 

• If degj^ a + dcg^ a > 0, then by item 2 we have 

sgn(a) = {-i f^s. a+dcg^T a sgn(ao), 
where uq = iL^i x"^^! the resuh of ehmmation of letters Xi,xf{l < i < t) from a. By 

items 3-8, Sgn(ao) ^\-lf''Sy^ ao+deg^r ao + l ^ (_l)dog^_r a+dcg^^ a+l_ 

Thus, 

Sgn(a) = ( — l^'^'^Sx a+dcg^T a+dcg^T a+dog_T a+1 

Formula (|23p concludes the proof. □ 

Lemma 7.12. Assume that (T, (-^^i, . . . , Xt,Yi, . . . ,Yr, Zi . . . , Zr)) is the tableau with substitution 
from Example \7.3\ and xi, . . . , Xt, yi, . . . , yr, zi . . . , Zr belong to Aiw- Then 

at':^{xi,...,xt,yi,...,yr,zi...,Zr) ^ ^ sgn(^) Yi ti{ip{a))\,^^.^ (^t+j)^y.^ (^t+r+j)^z,, 

where the substitution is applied for all 1 < i < t and 1 < J < In particular, 

...,Xt,Yi,...,Yr,Zi...,Zr)= bpf^(Xi, . . . , Xt,Yi, . . . ,Yr, Zi . . . , Zr) for n ^ t + 2r. 

Proof Using the fact that - TJi foi' ^ S t+2r if and only if ^ = r, we prove this lemma 
similar to Lemma 17.61 □ 

Remark 7.13. Lemma 17.111 can also be generalized for <yt,r,s{xi, . . . , Xu,yi, . . . , zi, . . . , Zw), 
where xi, . . . , z^ e Mw. 

We assume that crj ,.(^7 ^) stands for the element of JVa defined by Zubkov (see p. 292 of [24]). 
where t > 2r and x,y, z Adw- 

Lemma 7.14. For i, r > and x,y,z G M.w we have at^r{x, y, z) — a[j^2r,r{^j '^^ V)- 
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Proof. By Remark 14.81 we can assume that F = Q. Consider the tableau with substitution 
{T, {Xi, . . . , Xt,Yi, . . . ,Yr, Zi . . . , Zr)) from Example 17.31 By definition, we have 

a^_^_2^ ,.{x,z,y) ^ j^^—^ ^ sgn(cr) /(tr*(f7)), 

where tr*(cr) G is defined on p. 291 of [24] and / stands for the substitution i ^ Xi, (t+j) yj, 
{t + r + j) Zj for all 1 < z < i and I < j < t. Considering all 18 possibilities from the definition 
of tr* (ct) we can see that 

tr*(a)= n 

Lemmas 14.41 and 17.121 conclude the proof. □ 
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